This week I continued work on training squeezenet on the ImageNet dataset.
This day I continued letting squeezenet train on ImageNet. Sometime early Sunday there was an error that made its way into the savefile written after each epoch, which resulted in not just that instance of the script running crashing but every instance in the shell script I made crashing as well. I got it started up again, with hope it ran all the way this time. I also added a log file that tracks how the current training is doing, what the best so far was, and if it decided it should save. This can be compared with the 'best' save's modified time and see if at some point it was supposed to write and didn't or the other way around.
This day I continued training the model. I realized the values that determine if this model is the best so far (called 'prec1' and 'best_prec1') should not both always be 0.000000, as they are when training squeezenet. I ran a few epochs with alexnet to determine if this happens also with other networks, and it does. Odd. At least now I know what is happening.
This day I set to work finding the issue - it may be that the learning rate used was too high for squeezenet like how the default learning rate, while perfect for resnet, was too high for alexnet or VGG. I also increased the batch size (from 1 to 4) because that is part of the precision calculation, I suspect, and therefore increasing it allows the model greater diversity in percision beyond 0 and 100. It will be seen another day if this fixed it.
This day I discovered that did not fix it. Something with returning out of the validating method does not work (always returns 0), so this day was spent finding the bugs that keep the script from running validate (after running training for over an hour, which is frustrating) and then attempted to discover what the problem was.
Concurrently, I began designing the layout of my poster for the summer symposium next Thursday.
This day I did no work because I was on holiday.